Deidentifying Narrative Assessments to Facilitate Data Sharing in Medical Education.

PROBLEM: Narrative assessments are commonly incorporated into competency-based medical education programs. However, efforts to share competency-based medical education assessment data among programs to support the evaluation and improvement of assessment systems have been limited in part because of security concerns. Deidentifying assessment data mitigates these concerns, but deidentifying narrative assessments is time-consuming, resource intensive, and error prone. The authors developed and tested a tool to automate the deidentification of narrative assessments and facilitate their review. APPROACH: The authors met throughout 2021 and 2022 to iteratively design, test, and refine the deidentification algorithm and data review interface. Preliminary testing of the prototype deidentification algorithm was performed using narrative assessments from the University of Saskatchewan emergency medicine program. The algorithm's accuracy was assessed by the authors using the review interface designed for this purpose. Formal testing included 2 rounds of deidentification and review by members of the authorship team. Both the algorithm and data review interface were refined during the testing process. OUTCOMES: Authors from 3 institutions, including 3 emergency medicine programs, an anesthesia program, and a surgical program, participated in formal testing. In the final round of review, 99.4% of the narrative assessments were fully deidentified (names, nicknames, and pronouns removed). The results were comparable for each institution and specialty. The data review interface was improved with feedback obtained after each round of review and found to be intuitive. NEXT STEPS: This innovation has demonstrated viability evidence of an algorithmic approach to the deidentification of assessment narratives while reinforcing that a small number of errors are likely to persist. Future steps include the refinement of both the algorithm to improve its accuracy and the data review interface to support additional data set formats.

Developing a dashboard for program evaluation in competency-based training programs: a design-based research project.

Background: Canadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires the assessment of entrustable professional activities (EPAs). Dashboards could be used to track the completion of EPAs to support program evaluation. Methods: Using a design-based research process, we identified program evaluation needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. We interviewed leaders from the emergency medicine program and postgraduate medical education office at the University of Saskatchewan. Two investigators thematically analyzed interview transcripts to identify program evaluation needs that were audited by two additional investigators. Identified needs were described using quotes, analytics, and visualizations. Results: Between July 1, 2019 and April 6, 2021 we conducted 17 interviews with six participants (two program leaders and four institutional leaders). Four needs emerged as themes: tracking changes in overall assessment metrics, comparing metrics to the assessment plan, evaluating rotation performance, and engagement with the assessment metrics. We addressed these needs by presenting analytics and visualizations within a dashboard. Conclusions: We identified program evaluation needs related to EPA assessments and designed dashboard elements to meet them. This work will inform the development of other CBME assessment dashboards designed to support program evaluation.

Contexte: Les programmes de résidence dans une spécialité au Canada offrent une formation médicale fondée sur les compétences (FMFC) qui exige l'évaluation des activités professionnelles confiables (APC). Des tableaux de bord pourraient être utilisés pour suivre la complétion des APC afin de faciliter l'évaluation des programmes. Méthodes: Par un processus de recherche basé sur la conception, nous avons ciblé les besoins des programmes liés aux évaluations de la FMFC et conçu un tableau de bord qui comprend les éléments (données, analyses et visualisations) nécessaires pour répondre à ces besoins. Nous avons eu recours à des entretiens auprès des responsables du programme de médecine d'urgence et du bureau d'éducation médicale postdoctorale de l'Université de Saskatchewan. Deux enquêteurs ont effectué une analyse thématique des transcriptions des entretiens afin de recenser les besoins d'évaluation du programme, vérifiés par la suite par deux autres enquêteurs. Les besoins recensés ont été décrits à l'aide de citations, d'analyses et de visualisations. Résultats: Entre le 1er juillet 2019 et le 6 avril 2021, nous avons mené 17 entretiens avec six participants (deux responsables de programmes et quatre responsables de l'établissement). Quatre besoins sont ressortis en tant que thèmes : le suivi des changements dans les mesures d'évaluation globales, la comparaison des mesures avec le plan d'évaluation, l'évaluation de l'efficacité du stage et l'engagement face à des mesures d'évaluation. Nous avons répondu à ces besoins en présentant des analyses et des visualisations dans un tableau de bord. Conclusions: Nous avons identifié les besoins d'évaluation du programme liés aux évaluations des APC et conçu des éléments de tableau de bord pour y répondre. Ce travail guidera la conception d'autres tableaux de bord d'évaluation de la FMFC en vue de faciliter l'évaluation des programmes.

Developing a dashboard for faculty development in competency-based training programs: a design-based research project.

BACKGROUND: Canadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires frequent assessments of entrustable professional activities (EPAs). Faculty struggle to provide helpful feedback and assign appropriate entrustment scores. CBME faculty development initiatives rarely incorporate teaching metrics. Dashboards could be used to visualize faculty assessment data to support faculty development. METHODS: Using a design-based research process, we identified faculty development needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. Data was collected within the emergency medicine residency program at the University of Saskatchewan through interviews with program leaders, faculty development experts, and faculty participating in development sessions. Two investigators thematically analyzed interview transcripts to identify faculty needs that were audited by a third investigator. The needs were described using representative quotes and the dashboard elements designed to address them. RESULTS: Between July 1, 2019 and December 11, 2020 we conducted 15 interviews with nine participants (two program leaders, three faculty development experts, and four faculty members). Three needs emerged as themes from the analysis: analysis of assessments, contextualization of assessments, and accessible reporting. We addressed these needs by designing an accessible dashboard to present contextualized quantitative and narrative assessment data for each faculty member. CONCLUSIONS: We identified faculty development needs related to EPA assessments and designed dashboard elements to meet them. The resulting dashboard was used for faculty development sessions. This work will inform the development of CBME assessment dashboards for faculty.

CONTEXTE: Les programmes de résidence de spécialité au Canada mettent en œuvre une forme d'éducation axée sur les compétences (EASC) qui exige des évaluations formatives fréquentes des activités professionnelles confiables (APC). Les enseignants ont du mal à fournir une rétroaction utile et à attribuer des notes appropriées au niveau de confiance. Les initiatives de formation professorale des enseignants qui interviennent dans la EASC intègrent rarement leurs données psychométriques. Des tableaux de bord pourraient être utilisés pour visualiser les données d'évaluation du corps professoral afin de soutenir leur perfectionnement. MÉTHODES: En utilisant un processus de recherche orientée par la conception, nous avons déterminé les besoins de formation professorale liés aux évaluations dans la EASC et nous avons conçu un tableau de bord contenant des éléments (données, analyses et éléments visuels) pour répondre à ces besoins. Les données ont été recueillies dans le cadre du programme de résidence en médecine d'urgence de l'Université de Saskatchewan par le biais d'entretiens avec les responsables du programme, des experts en formation professorale et les enseignants participant aux séances de formation. Deux chercheurs ont procédé à une analyse thématique des transcriptions d'entrevues afin d'identifier les besoins des enseignants, et un troisième chercheur les a vérifiées. Les besoins ont été décrits à l'aide de citations représentatives et des éléments du tableau de bord conçus pour y répondre. RÉSULTATS: Entre le 1er juillet 2019 et le 11 décembre 2020, nous avons mené 15 entretiens avec neuf participants (deux responsables de programme, trois experts en formation professorale et quatre membres du corps professoral). Trois besoins sont ressortis comme thèmes de l'analyse : l'analyse des évaluations formatives, la contextualisation des évaluations formatives et l'accessibilité des rapports. Pour répondre à ces besoins, nous avons conçu un tableau de bord accessible présentant des données d'évaluation quantitatives et narratives contextualisées pour chaque membre du corps professoral. CONCLUSIONS: Nous avons identifié les besoins de formation professorale liés aux évaluations des APC et conçu les éléments d'un tableau de bord permettant d'y répondre. Le tableau de bord a été utilisé dans des séances de formation professorale. Ce travail orientera la réalisation de tableaux de bord afin de faciliter l'évaluation pour les enseignants dans le cadre de la EASC.

CAEP 2019 Academic Symposium: Got competence? Best practices in trainee progress decisions.

BACKGROUND: Competence committees play a key role in a competency-based system of assessment. These committees are tasked with reviewing and synthesizing clinical performance data to make judgments regarding residents' competence. Canadian emergency medicine (EM) postgraduate training programs recently implemented competence committees; however, a paucity of literature guides their work. OBJECTIVE: The objective of this study was to develop consensus-based recommendations to optimize the function and decisions of competence committees in Canadian EM training programs. METHODS: Semi-structured interviews of EM competence committee chairs were conducted and analyzed. The interview guide was informed by a literature review of competence committee structure, processes, and best practices. Inductive thematic analysis of interview transcripts was conducted to identify emerging themes. Preliminary recommendations, based on themes, were drafted and presented at the 2019 CAEP Academic Symposium on Education. Through a live presentation and survey poll, symposium attendees representing the national EM community participated in a facilitated discussion of the recommendations. The authors incorporated this feedback and identified consensus among symposium attendees on a final set of nine high-yield recommendations. CONCLUSION: The Canadian EM community used a structured process to develop nine best practice recommendations for competence committees addressing: committee membership, meeting processes, decision outcomes, use of high-quality performance data, and ongoing quality improvement. These recommendations can inform the structure and processes of competence committees in Canadian EM training programs.

CAEP 2016 Academic Symposium on Education Scholarship: Training our Future Clinician Educators in Emergency Medicine.

OBJECTIVE: To develop consensus recommendations for training future clinician educators (CEs) in emergency medicine (EM). METHODS: A panel of EM education leaders was assembled from across Canada and met regularly by teleconference over the course of 1 year. Recommendations for CE training were drafted based on the panel's experience, a literature review, and a survey of current and past EM education leaders in Canada. Feedback was sought from attendees at the Canadian Association of Emergency Physicians (CAEP) annual academic symposium. Recommendations were distributed to the society's Academic Section for further feedback and updated by a consensus of the expert panel. RESULTS: Recommendations were categorized for one of three audiences: 1) Future CEs; 2) Academic departments and divisions (AD&D) that support training to fulfill their education leadership goals; and 3) The CAEP Academic Section. Advanced medical education training is recommended for any emergency physician or resident who pursues an education leadership role. Individuals should seek out mentorship in making decisions about career opportunities and training options. AD&D should regularly perform a needs assessment of their future CE needs and identify and encourage potential individuals who fulfill education leadership roles. AD&D should develop training opportunities at their institution, provide support to complete this training, and advocate for the recognition of education scholarship in their institutional promotions process. The CAEP Academic Section should support mentorship of future CEs on a national scale. CONCLUSION: These recommendations serve as a framework for training and supporting the next generation of Canadian EM medical educators.

CAEP 2016 Academic Symposium: How to have an impact as an emergency medicine educator and scholar.

BACKGROUND: In a time of major medical education transformation, emergency medicine (EM) needs to nurture education scholars who will influence EM education practice. However, the essential ingredients to ensure a career with impact in EM education are not clear. OBJECTIVE: To describe how to prepare EM educators for a high-impact career. METHODS: The Canadian Association of Emergency Physicians (CAEP) Academic Section commissioned an "Education Impact" working group (IWG) to guide the creation of consensus recommendations from the EM community. EM educators from across Canada were initially recruited from the networks of the IWG members, and additional educators were recruited via snowball sampling. "High impact educators" were nominated by this network. The high impact educators were then interviewed using a structured question guide. These interviews were transcribed and coded for themes using qualitative methods. The process continued until no new themes were identified. Proposed themes and recommendations were presented to the EM community at the CAEP 2016 Academic Symposium. Feedback was then incorporated into a final set of recommendations. RESULTS: Fifty-five (71%) of 77 of identified Canadian EM educators participated, and 170 names of high impact educators were submitted and ranked by frequency. The IWG achieved sufficiency of themes after nine interviews. Five recommendations were made: 1) EM educators can pursue a high impact career by leveraging either traditional or innovative career pathways; 2) EM educators starting their education careers should have multiple senior mentors; 3) Early-career EM educators should immerse themselves in their area of interest and cultivate a community of practice, not limited to EM; 4) Every academic EM department and EM teaching site should have access to an EM educator with protected time and recognition for their EM education scholarship; and 5) Educators at all stages should continuously compile an impact portfolio. CONCLUSIONS: We describe a unique set of recommendations to develop educators who will influence EM, derived from a consensus from the EM community. EM leaders, educators, and aspiring educational scholars should consider how to implement this guide towards enhancing our specialty's educational mission.

Administration and leadership competencies: establishment of a national consensus for emergency medicine.

INTRODUCTION: The Royal College of Physicians and Surgeons of Canada requires emergency medicine (EM) residency programs to meet training objectives relating to administration and leadership. The purpose of this study was to establish a national consensus on the competencies for inclusion in an EM administration and leadership curriculum. METHODS: A modified Delphi process involving two iterative rounds of an electronic survey was used to achieve consensus on competencies for inclusion in an EM administration and leadership curriculum. An initial list of competencies was compiled using peer-reviewed and grey literature. The participants included 14 EM residency program directors and 43 leadership and administration experts from across Canada who were recruited using a snowball technique. The proposed competencies were organized using the CanMEDS Physician Competency Framework and presented in English or French. Consensus was defined a priori as >70% agreement. RESULTS: Nearly all (13 of 14) of the institutions with an FRCPC EM program had at least one participant complete both surveys. Thirty-five of 57 (61%) participants completed round 1, and 30 (53%) participants completed both rounds. Participants suggested an additional 16 competencies in round 1. The results of round 1 informed the decisions in round 2. Fifty-nine of 109 (54.1%) competencies achieved consensus for inclusion. CONCLUSIONS: Based on a national modified Delphi process, we describe 59 competencies for inclusion in an EM administration and leadership curriculum that was arranged by CanMEDS Role. EM educators may consider these competencies when designing local curricula.

Teaching the RAPID approach at the start of emergency medicine clerkship: an evaluation.

BACKGROUND: The RAPID approach (Resuscitation, Analgesia and assessment, Patient needs, Interventions, Disposition) was developed as an approach to managing emergency department patients. It is a mental checklist to help trainees provide comprehensive care, addressing issues in priority. Its impact on trainee performance has not been assessed. METHODS: Forty-two clerkship students were enrolled, with 21 students in each group. They received or did not receive the teaching intervention on an alternate basis. Students were assessed through daily encounter cards, a case presentation, a self-assessment form, a prerotation case (case E), and a six-case short-answer exit examination (cases A to F) with case E repeated. Case E was designed specifically to assess students' ability to provide comprehensive care. Fourteen students participated in focus groups. RESULTS: Students in the intervention group had significantly higher exit examination case E scores (11.67 of 14 v. 10.26 of 14, p  =  0.008) and improvement in their case E scores from pre- to postrotation (1.82 v. 0.26, p  =  0.006). There were no significant differences in the other outcome measures. Intervention group students made positive comments around analgesia, addressing nonmedical needs and counseling on health promotion during focus groups. CONCLUSIONS: Students exposed to the RAPID approach at the start of their emergency medicine rotation performed better on the one component of the written examination for which it was designed to improve performance. Students found it to be a useful mental checklist for comprehensive care, possibly addressing the hidden curriculum. Emergency medicine educators should consider further study and careful implementation of the RAPID approach.

Core competencies for emergency medicine clerkships: results of a Canadian consensus initiative.

OBJECTIVES: There is no consensus on what constitutes the core competencies for emergency medicine (EM) clerkship rotations in Canada. Existing EM curricula have been developed through informal consensus and often focus on EM content to be known at the end of training rather than what is an appropriate focus for a time-limited rotation in EM. We sought to define the core competencies for EM clerkship in Canada through consensus among an expert panel of Canadian EM educators. METHODS: We used a modified Delphi method and the CanMEDS 2005 Physician Competency Framework to develop a consensus among expert EM educators from across Canada. RESULTS: Thirty experts from nine different medical schools across Canada participated on the panel. The initial list consisted of 152 competencies organized in the seven domains of the CanMEDS 2005 Physician Competency Framework. After the second round of the Delphi process, the list of competencies was reduced to 62 (59% reduction). A complete list of competencies is provided. CONCLUSION: This study established a national consensus defining the core competencies for EM clerkship in Canada.

Take my breath away: a case of lactic acidosis in an asthma exacerbation.

A 36-year-old male with a history of chronic asthma presented to an emergency department with shortness of breath consistent with an asthma exacerbation. He had persistent tachypnea following inhaled bronchodilator treatment; thus, the workup and differential diagnosis were expanded. He was found to have a mixed respiratory alkalosis and metabolic acidosis with elevated serum lactate without an obvious cause and was admitted to hospital. His case was reviewed, and the lactic acidosis was thought to be caused by inhaled ß2-agonist use. Emergency physicians should be aware of the potential side effects of inhaled ß2-agonists as lactic acidosis may complicate clinical assessment and management of asthma exacerbations and lead to unnecessary and potentially dangerous escalations in therapy.

Using a Delphi process to establish consensus on emergency medicine clerkship competencies.

BACKGROUND: Currently, there is no consensus on the core competencies required for emergency medicine (EM) clerkships in Canada. Existing EM curricula have been developed through informal consensus or local efforts. The Delphi process has been used extensively as a means for establishing consensus. AIM: The purpose of this project was to define core competencies for EM clerkships in Canada, to validate a Delphi process in the context of national curriculum development, and to demonstrate the adoption of the CanMEDS physician competency paradigm in the undergraduate medical education realm. METHODS: Using a modified Delphi process, we developed a consensus amongst a panel of expert emergency physicians from across Canada utilizing the CanMEDS 2005 Physician Competency Framework. RESULTS: Thirty experts from nine different medical schools across Canada participated on the panel. The initial list consisted of 152 competencies organized in the seven domains of the CanMEDS 2005 Physician Competency Framework. After the second round of the Delphi process, the list of competencies was reduced to 62 (59% reduction). CONCLUSION: This study demonstrated that a modified Delphi process can result in a strong consensus around a realistic number of core competencies for EM clerkships. We propose that such a method could be used by other medical specialties and health professions to develop rotation-specific core competencies.

Consultation outcomes in the emergency department: exploring rates and complexity.

OBJECTIVE: Consultation is a common and important aspect of emergency department (ED) care. We prospectively examined the consultation rates, the admission rates of consulted patients, the emergency physician (EP) disposition prediction of consulted patients and the difficult consultations rates in 2 tertiary care hospitals. METHODS: Attending EPs recorded consultations during 5 randomly selected shifts over an 8-week period using standardized forms. Subsequent computer outcome data were extracted for each patient encounter, as well as demographic data from the ED during days in which there was a study shift. RESULTS: During 105 clinical shifts, 1930 patients were managed by 21 EPs (median 17 patients per shift; interquartile range 14-23). Overall, at least 1 consultation was requested in 38% of patients. More than one-half of the patients (54.3%) who received a consultation were admitted to the hospital. Consultation proportions were similar between males and females (51% v. 49%, p=0.03). Consultations occurred more frequently for patients who were older, had higher acuity presentations, arrived during daytime hours or arrived by ambulance. The proportion of agreement between the EP's and consultant's opinion on the need for admission was 89% (kappa=0.77, 95% confidence interval 0.72-0.83). Overall, 92% of patents received 1 consultation. Six percent of the consultations were perceived as "difficult" by the EPs (defined as the EP's subjective impression of difficulties with consultation times, accessibility and availability of consultants, and the interaction with consultants or disposition issues). CONCLUSION: Consultation is a common process in the ED. It often results in admission and is predictable based on simple patient factors. Because of perceived difficulty with consultations, strategies to improve the EP consultation process in the ED seem warranted.

Brugada syndrome in a young patient with palpitations.

Brugada syndrome is a potentially lethal and eminently treatable entity that may present with palpitations or syncope. This article presents the case of a young patient with Brugada syndrome and reviews key features in the epidemiology, pathophysiology, diagnosis, treatment and prognosis of this condition.